Drawing on the World: Sketch in Context

In this paper we introduce our approach to implementing context-rich sketch-based interfaces. By “contextrich” we mean interfaces for systems that refer to real-world objects. For example, a system that allowed the user to draw on an annotated video feed instead of a blank canvas would yield this kind of context-richness. We describe “Drawing on the World”, the concept that taking what is being drawn on into consideration results in increased ease of development and a better user experience in sketch recognition. We created a description language, called StepStool, and an engine to interpret StepStool descriptions. StepStool is used to describe the relationship between context (objects on the canvas) and drawn shapes to determine the shapes’ meanings. We used StepStool to implement the context-rich control interface to a robotic forklift (see [2], [11]). We use that interface in this paper to describe StepStool’s use. In our future work, we propose two extensions to StepStool that could make it more broadly applicable. The first extension allows StepStool to be used with non-robotic systems. The second proposed extension allows StepStool to be used with other modalities—e.g., hand gesture recognition.